When Will It Be Done? Machine Learner Answers to the 300-Billion-Dollar Question

W hen will it be done? " Senior managers will ask their software project managers this question more than 250,000 times this year. Corporations, which collectively commit over US$300 billion annually toward new software project initiatives, 1 will want to know the answer. However, when you consider Barry Boehm's claim that early software life-cycle estimates vary by a factor of four (25 to 400 percent), 2 providing an accurate, reliable project estimate presents a challenge indeed. To answer the question, a project manager might resort to one of three estimation approaches: human-based, algorithmic, or machine learner-based. Managers will use a human-based approach—which includes expert judgment, analogy, and rule of thumb—87 percent of the time, algorithmic or machine learners only 13 percent of the time. 3 Considering the popularity of algorithmic models (such as Function Point Analysis) for estimating effort, machine learner use is probably much less than half of 13 percent—perhaps three or four percent. This is surprising considering recent successes in machine learner estimation. In three instances, machine learners produced project effort estimates within 25 percent accuracy at least 75 percent of the time. The high-water mark of these efforts produces estimates with 25 percent accuracy 83 percent of the time. 4 All three cases greatly exceed Boehm's factor-of-four expectation. I contend that a greater application of AI, through machine learners , in software estimation would greatly improve accuracy, reliability, and repeatability in software development. To standardize results, I define two benchmarks for the case studies: • Prediction accuracy in project estimation refers to how often a machine learner can achieve a result within a specified range. The informal standard in effort estimation is 25 percent accuracy, denoted as pred(0.25). So, a result of pred(0.25) = 75 percent means the machine learner model produces results with 25 percent accuracy 75 percent of the time. • Mean magnitude of relative error is the average difference between actual and calculated results in absolute terms. Many consider an MMRE below 25 percent a good model and below 50 percent a usable model. All three cases assume an accuracy rate of pred(0.25) and build their machine learner-based models using data extracted from industrial settings , as opposed to deriving models from simulations. This reinforces these results' real-world applicability. development effort by applying a back-propagation neural network model to the Australian Software Metrics Association data set. 5 You can obtain the ASMA data set …